Intersection infrastructure warning system

ABSTRACT

A warning system for warning vulnerable road users (VRUs) includes an edge computing device having a graphics processing unit (GPU), one or more sensors that acquire spatial-temporal data of road users, and one or more warning devices that output a warning to warn targeted VRUs of danger. The GPU uses one or more artificial intelligence (AI) algorithms to process the spatial-temporal data of the road users to predict a path, trajectory, behavior, and intent for the road users. The GPU then analyzes the predictions of the road users to determine convergences between the predictions to determine threat interactions and identify targeted VRUs. In response to determining threat interactions and identifying the targeted VRUs, the edge computing device outputs targeting instructions and warning response instructions to the one or more warning devices to deploy the one or more warning devices to warn the targeted VRUs.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/949,022 filed Dec. 17, 2019. The entire disclosure of the applicationreferenced above are incorporated by reference.

FIELD

The present disclosure relates to intersection warning systems. Morespecifically, the present disclosure relates to intersectioninfrastructure for providing warnings to vulnerable road users at ornear intersections.

BACKGROUND

The background description provided here is for the purpose of generallypresenting the context of the disclosure. Work of the presently namedinventors, to the extent it is described in this background section, aswell as aspects of the description that may not otherwise qualify asprior art at the time of filing, are neither expressly nor impliedlyadmitted as prior art against the present disclosure.

Conventional intersection infrastructure can warn both vulnerable roadusers (VRUs), (e.g. pedestrians) and vehicle drivers of conditionsconducive to a collision between a VRU and a vehicle at a trafficintersection. Conventional intersection infrastructure included inintelligent transportation system (ITS) with vehicle-to-infrastructure(V2I) communication systems and vehicles equipped with dedicatedshort-range communications (DSRC) as part of the V2I system mayrespectively send and receive warning messages to alert a vehicle driverof potential collisions between a VRU at an intersection. Conventionalintersection infrastructure may also be able to send text warnings tothe cellular phones of VRUs around the intersection via a short messageservice (SMS) or dedicated mobile application software (i.e., mobileapp) to warn the VRU of potential dangers at an intersection.Conventional intersection infrastructure may also provide graphicalwarnings and warning messages to an external display visible to bothVRUs and vehicle drivers around the intersection. However, whileconventional intersection infrastructure can provide some warning tolimited VRUs and vehicle drivers around an intersection, theconventional intersection infrastructure for warning VRUs of possibledanger from vehicles around the intersection is subject to improvement.

SUMMARY

In one example, an intersection infrastructure warning system forwarning vulnerable road users (VRUs) of danger around an intersectionmay include an edge computing device having a graphics processing unit(GPU), one or more sensors configured to acquire spatial-temporal dataof road users in a 360° area around the intersection, and one or morewarning devices configured to output a warning to warn targeted VRUs ofthe danger around the intersection. The GPU may be configured to use oneor more artificial intelligence (AI) algorithms to process thespatial-temporal data of the road users to determine at least one ofpath, trajectory, behavior, and intent predictions for the road users.The GPU may be further configured to analyze the predictions of the roadusers to determine convergences between the predictions of the roadusers to determine threat interactions and identify targeted VRUs. Inresponse to determining threat interactions and identifying the targetedVRUs, the edge computing device may be further configured to outputtargeting instructions and warning response instructions to the one ormore warning devices to deploy the one or more warning devices to thetargeted VRUs. In response to receiving the targeting instructions andwarning response instructions, the one or more warning devices may befurther configured to target the targeted VRUs based on the targetinginstructions and to output the warning to the targeted VRUs.

In another example, a method for warning VRUs of danger around anintersection may include identifying road users around the intersectionand accumulating spatial-temporal data for each of the identified roadusers. The method may further include processing the spatial-temporaldata for each of the identified road users to determine at least one oftrajectories, paths, intents, and spatial-temporal behaviors for each ofthe identified road users and processing the at least one of thetrajectories, paths, intents, and spatial-temporal behaviors for each ofthe identified road users using a first artificial intelligencealgorithm to determine at least one of predicted trajectories, predictedpaths, predicted intents, and predicted spatial-temporal behaviors. Themethod may further include analyzing the predictions for each of theidentified road users using a second AI algorithm to determine if anyconflict zones exist for any of the identified road users, the conflictzones indicating an interaction risk between any of the identified roadusers.

In even another example, an intersection infrastructure warning systemfor warning VRUs of danger around an intersection may include an edgecomputing device having a GPU, one or more sensors configured to acquiredata for determining spatial-temporal data of road users in a 360° areaaround the intersection, and one or more drones configured to output awarning to warn targeted VRUs of the danger around the intersection. TheGPU may be configured to use one or more A algorithms to process thespatial-temporal data of the road users to predict at least one of apath, a trajectory, a behavior, and an intent of a road user. The GPUmay be further configured to analyze the predictions of the road usersto determine convergences between the predictions of the road users todetermine threat interactions and identify the targeted VRUs. Inresponse to determining threat interactions and identifying the targetedVRUs, the edge computing device may be further configured to outputtargeting instructions and warning response instructions to the one ormore drones to deploy the one or more drones to the targeted VRUs. Andin response to receiving the targeting instructions and the warningresponse instructions from the edge computing device, the one or moredrones may be further configured to fly to the targeted VRUs based onthe targeting instructions and to output the warning to the targeted VRUbased on the warning response instructions.

Further areas of applicability of the present disclosure will becomeapparent from the detailed description, the claims, and the drawings.The detailed description and specific examples are intended for purposesof illustration only and are not intended to limit the scope of thedisclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure will become more fully understood from thedetailed description and the accompanying drawings.

FIG. 1 is a block schematic of an intersection infrastructure warningsystem;

FIG. 2 is a perspective view of a pan tilt device;

FIG. 3 is a block schematic of a drone and a nest;

FIG. 4 illustrates a perspective view of an intersection using anintersection infrastructure warning system;

FIG. 5 is a flowchart illustrating a process flow for processesperformed by an edge computing device;

FIG. 6 is an example warning response matrix;

FIG. 7 is a perspective view of an intersection infrastructure warningsystem illustrating an example operation of one example embodiment;

FIG. 8 is a perspective view of an intersection infrastructure warningsystem illustrating another example operation for another exampleembodiment;

FIG. 9 is a perspective view of an intersection infrastructure warningsystem illustrating another example operation for another exampleembodiment;

FIG. 10 is a perspective view of an intersection infrastructure warningsystem illustrating another example operation for another exampleembodiment; and

FIG. 11 is a perspective view of an intersection infrastructure warningsystem illustrating another example operation for another exampleembodiment.

In the drawings, reference numbers may be reused to identify similarand/or identical elements.

DETAILED DESCRIPTION

In conventional intersection infrastructure for providing warnings ofconditions conducive to a collision between a vulnerable road user (VRU)and a motor vehicle, the conventional systems may only use data such asbasic safety messages (BSM) from vehicles equipped with dedicatedshort-range communications (DSRC) as part of a vehicle-to-infrastructure(V2I) system for communicating with the infrastructure to determine thepossibility of a collision between a VRU and the connected motorvehicle. In other words, these conventional systems rely on onlyconnected vehicles equipped with DSRC to determine potential collisionswith VRUs.

Moreover, conventional systems may lack the sensors necessary forproviding a complete 360° view of the intersection and the computationalcapacity to predict the trajectories of motor vehicles and VRUs aroundthe intersection based on the sensor data. That is, conventional systemsdo not provide a full cognitive sphere or full intersection vicinityanalysis of threat conditions to the VRUs, as well as determiningprecise locations, trajectories, and paths of the VRUs and the vehicles.Such gaps in analysis can result in missed threat warnings to the VRUsthat may result in vehicle collisions causing injury. The analysis gapsmay also cause false positive warnings that may diminish the VRUs' trustin the warning system and cause VRUs to ignore potential warnings, evenif legitimate.

For conventional systems that warn VRUs via the VRUs' mobile phones,such systems assume that VRUs will be carrying mobile phones, that themobile phones include the necessary hardware/software for receivingwarning messages, and that the connected VRUs are timely notified ofsuch collision threats. That is, conventional systems may rely on VRUsbeing connected to the system via mobile phones and assume that the VRUscan actually be alerted by their mobiles phones in time to perceive thethreat and take remedial action. However, in reality, VRUs may miss suchalerts resulting in injury. Mobile phones also have limited computingand bandwidth capabilities. As such, even if such phones are networkedto infrastructure and/or networked with other users via mobileapplications for providing/receiving traffic updates and trafficwarnings, such arrangements are limited by the computing and bandwidthcapabilities of the mobile phone and have inherent latencies, limitedsensor coverage, and recurrent communications between the mobile phonesand infrastructure resulting in limited reaction times for the VRUs foraccident mitigation and/or prevention.

In conventional systems that provide a more personalized warning to aVRU, such as by use of a drone, the drones of conventional systems mayprocess all the computations necessary for warning a road user on-boardthe drone itself. That is, the drone itself may use on-board sensors toidentify a threat to warn a VRU. However, the on-board computational andsensing capabilities of conventional drone warning systems are limited.That is, conventional drones lack enhanced computational and sensingcapabilities. The lack of enhanced processing power and advanced sensorsin conventional drones can cause analysis gaps due to the limitedprocessing power and sensing capabilities of such systems. The gaps inanalyzing predicted trajectories and real-time paths of all road usersaround the intersection may cause missed warnings to VRUs and thesemissed warnings may result in injury to the VRUs. Alternatively, or inaddition to the missed warnings, the analysis gaps caused byconventional drones may also cause false-positive warnings, where thedrones erroneously alert VRUs to dangerous situations that will neveroccur. Accordingly, systems with too many erroneous warnings may causethe VRUs to ignore the warnings altogether, even if legitimate. Thus,the drones in conventional systems may not provide complete access andfull views of the vicinity surrounding the intersection. Additionally,the drones of conventional systems may not be able to analyze multipleconcurrent scenarios to provide broader safety.

Example embodiments are described with reference to the accompanyingdrawings.

With reference to FIG. 1, a block schematic of an intersectioninfrastructure warning system 1 for providing warnings to vulnerableroad users (VRUs) around an intersection is shown. VRUs may refer topedestrians and cyclists around an intersection while “road users” maydenote all road users including motor vehicles, which may be referred tosimply as “vehicles.” “At” or “around” the intersection may denote areasaround the intersection such as sidewalks, junction roads leading intothe intersection, the intersection itself (i.e., the inside area of theintersection where the junction roads can no longer be discerned fromone another), and the surrounding vicinity.

The intersection infrastructure 1 includes an edge computing device 10,one or more sensors in a sensor array 50, and a warning device 60.

As compared to conventional computing systems, the edge computing device10 has enhanced processing capabilities, lower latency, and fasterresponse times. Based on these enhanced capabilities, the edge computingdevice 10 of the intersection infrastructure 1 can apply artificialintelligence (AI) algorithms to sensor data to determine a 360° analysisof an intersection and the surrounding vicinity to calculate the preciselocations and paths of road users around the intersection, and calculatethe predicted trajectories and paths of the road users around theintersection. Based on these calculations, the edge computing device 10can determine multiple concurrent risk scenarios using AI algorithms andmodels to better identify at-risk VRUs while also identifying thevehicles posing the risk. By using the A algorithms, the edge computingdevice 10 of the intersection infrastructure 1 can increase theconfidence level of the calculated predictions. The faster computationalcapabilities of the edge computing device 10 can increase the timehorizon for warning a VRU to provide better real-time warnings to VRUs,mitigate danger, and increase safety margins around the intersection.The calculations by the edge computing device 10 can be used as thebasis for outputting instructions to a warning device 60 to better warn,intercept, or direct one or more VRUs at the intersection in real-timeto mitigate collisions between the VRUs and the vehicles around theintersection.

The edge computing device 10 is configured as a distributed computingsystem that includes a road side unit (RSU) 20 that networks andcommunicates with a distributed cloud networking system 40 (i.e., “thecloud”). The RSU 20 includes a graphics processing unit (GPU) 22, acentral processing unit (CPU) 24, storage 26, and a communicationsmodule 30. The RSU 20 may be housed inside a traffic cabinet 21 at anintersection. The traffic cabinet 21 may include other hardware inaddition to the RSU 20 for controlling the traffic signals at anintersection. The RSU 20 of the edge computing device 10, the sensorarray 50, and the warning device 60 may be powered directly and/orindirectly from mains electricity used for powering other electriccomponents at the intersection such as the control signals, thepedestrian signals, speakers for audible signals, street lights,electric signage, traffic control signal hardware, and the like. Thatis, the intersection infrastructure warning system 1 may be powered bythe electric infrastructure already in place at the intersection. Whilethe RSU 20 of the edge computing device 10 may be part of avehicle-to-infrastructure (V2I) system, the RSU 20 of the presentdisclosure differs from conventional RSUs, in that the RSU 20 includesenhanced computational abilities for executing parallel computationsusing AI algorithms. The RSU 20 may also be a traditional intelligenttransportation system (ITS) V2X device that is configured to coordinatewith the edge computing device 10.

The GPU 22 includes various interfaces such as a bus interface and adisplay interface, a video processing unit (VPU), a graphics memorycontroller (GMC), a compression unit, and a graphics and computer array(GCA), among other components (all not shown). The GPU 22 supportsmassive threading and parallel computing and is a CUDA-enabled GPU. CUDAis an abbreviation for Compute Unified Device Architecture and is aregistered trademark of the Nvidia Corporation. CUDA is a parallelcomputing platform and application programming interface that allows theGPU 22 to be used for general purpose parallel processing. While CUDA isused as an example to support parallel computing, the GPU 22 may use analternative platform and API for parallel processing.

By using the GPU 22, large blocks of data can be used in calculationswith AI algorithms more effectively and efficiently than the samecalculations using the CPU 24. Such AI algorithms may include, forexample, an unsupervised learning algorithm like an Isolated Forestmodel and a deep learning algorithm like a Sequence-to-sequence model.In other words, using the GPU 22 allows the edge computing device 10 tomore quickly execute parallel calculations using AI algorithms toanalyze and process measurements from the sensor array 50 and to predictmultiple concurrent risk scenarios for vehicles and VRUs around anintersection based on the measurements from the sensor array 50. The GPU22 may be used with A algorithms to process and analyze measurement datafrom the sensor array 50, as well as other data, to determinespatial-temporal data and behaviors for each of the road users, and thenpredict the paths, trajectories, intent, and behavior for all road usersaround an intersection, including vehicles. The spatial-temporal dataand predictions determined by the GPU 22 for each of the road users maybe stored in the storage 26. The spatial-temporal data and predictionsby the GPU 22 may be sent to the distributed cloud networking system 40via the communication module 30 for further processing, such as modelingand simulation.

The CPU 24 may be a processor for executing less computational intensiveprograms and instruction sets than the GPU 22. The CPU 24 may also beconfigured as a microcontroller or as a System on Chip (SoC). Forexample, the CPU 24 may execute programs and instruction sets fortransferring data between the storage 26, the GPU 22, and thecommunication module 30. The CPU 24 may also be used for controlling thecommunication module 30 to transfer and receive data from thedistributed cloud networking system 40.

The CPU 24 may also be used as an input/output for receiving andtransmitting data to/from the sensor array 50 and the warning device 60.Alternatively, the communication module 30 may be used forcommunications between the RSU 20, the sensor array 50, and the warningdevice 60.

The storage 26 may be a memory such as RAM, ROM, and flash memory,and/or a storage device such as a magnetic hard drive (HDD) or asolid-state drive (SSD) using flash memory. The storage 26 may be usedto store spatial-temporal data of the road users and pre-trained modelsused by the A algorithms executed by the GPU 22. The storage 26 may alsostore prediction data from the GPU 22 for further processing by thedistributed cloud networking system 40 to generate trained models andrun simulations. The storage 26 may also store programs, instructionsets, and software used by the GPU 22 and the CPU 24.

The communication module 30 allows the RSU 20 of the edge computingdevice 10 to transmit and receive signals and data with externalsystems, devices, and networks. Generally, the communication module 30may be used to input and output signals and data to and from the RSU 20.The communication module 30 may be used to receive messages from otherconnected infrastructure such as signal phase and timing (SPaT) messagesfrom traffic and pedestrian control signals, basic safety messages(BSMs) from vehicles having dedicated short-range communication andconnected to a vehicle-to-everything (V2X) system, and personal safetymessages (PSMs) from pedestrians and cyclists connected to the V2Xsystem (e.g., by a mobile phone). The communication module 30 may alsobe used to broadcast SPaT messages and intersection MAP messages toconnected road users.

The communication module 30 may include a wireless access point (WAP)32, gateway, or like networking hardware to wirelessly connect the RSU20 to an external network such as a wireless local area network (WLAN)or LAN. For example, the wireless access point 32 may be configured tocommunicate wirelessly using an IEEE 802.11 protocol. Alternatively, orin addition to the WAP 32, the communication module 30 may include atransmitting and receiving device 34 that is configured to communicateeither wirelessly or by wire with external devices. The transmitting andreceiving device 34 may be, for example, a transceiver, a modem, and anetwork switch. For example, the transmitting and receiving device 34may be a cellular transceiver 34 configured to transmit and receivecellular signals at cellular allocated frequencies. As such, thecellular transceiver 34 may be configured for mobile telecommunicationand cellular network technologies such as 2G, 3G, 4G LTE, and 5G fortransmitting and receiving data to provide mobile broadband capabilitiesto the RSU 20. A cellular transceiver 34 can connect the RSU 20 to awireless wide area network (WWAN) or WAN. Generally, the communicationmodule 30 may be configured for wired and wireless communications usingcommon communication standards and technology such as IEEE 802.3, IEEE802.11, Bluetooth, mobile broadband, and the like.

The communication module 30 may be connected by wired connection orwirelessly with the sensors of the sensor array 50 and the one or morewarning devices 60. The communication module 30 may also include one ormore antennas 36 for transmitting radio signals from the communicationmodule 30 and receiving radio signals at the communication module 30.Alternatively, both the WAP 32 and the transmitting and receiving device34 may respectively include one or more individual antennas.

The distributed cloud networking system 40 (i.e., “the cloud”) is one ormore cloud computing elements that is part of the edge computing device10. The distributed cloud networking system 40 provides additionalresources like data storage and processing power to the edge computingdevice 10. Because the distributed cloud networking system 40 isaccessible over the Internet, the cloud 40 is configured to communicatewith the RSU 20 of the edge computing device 10 via the communicationmodule 30.

The cloud 40 may include any number or different services such asinfrastructure as a service (IaaS), platform as a service (PaaS),software as a service (SaaS), backend as a service (BaaS), serverlesscomputing, and function as a service (FaaS). The cloud 40 may be acommercial cloud computing service such as Amazon Web Services (AWS),Microsoft Azure, Google Cloud Platform (GCP), or Oracle Cloud, allregistered trademarks.

In addition to the AI algorithms used by the GPU 22 to calculate thepredicted trajectories and paths of road users around an intersection,the cloud 40 may be used to calculate trained models and run simulationsthat can be used by the GPU 22 and applied to the A algorithms to bettercalculate the predicted paths, trajectories, and behaviors of the roadusers around an intersection, and to better identify the vehicles posingthreats to the VRUs. Trained models calculated by the cloud 40 may bestored in the storage 26 of the RSU 20 for use by the GPU 22.

While the description may describe specific components of the edgecomputing device 10 performing a function or process, the edge computingdevice 10 generally performs the same function or process described asbeing performed by the subsystem or sub-component of the edge computingdevice 10. That is, higher level components can also be described asperforming the same functions as their subsystems and sub-components.For example, while the GPU 22 is described as performing calculationsusing AI algorithms, both the edge computing device 10 and the RSU 20can also be described as performing calculations using AI algorithms.

The sensor array 50 includes sensors that are used to acquirespatial-temporal data from all road users around the intersection,including vehicles. The sensor data from the sensor array 50 can be usedby the GPU 22 to calculate the predicted paths and trajectories of allroad users around an intersection. That is, the sensor array 50 acquiresspatial-temporal data for detecting, localizing, and tracking all roadusers around the intersection. The intersection infrastructure mayinclude one or more sensor arrays 50 at different locations around anintersection to obtain a 360° view and sensing area of the intersection.The one or more sensor arrays 50 at an intersection may provide aviewing and sensing area, for example, with a two hundred meter radiuscentered at the intersection. That is, the sensors 52 and 54 in thesensor array 50 have a range of about two hundred meters from theintersection.

Each sensor array 50 may include one or more cameras 52 and one moredetection and ranging sensors 54. While the camera 52 and the detectionand ranging sensor 54 are described as being part of a sensor array 50,the camera 52 and the detecting and ranging sensor 54 are notnecessarily limited to this configuration and may be disposed separatelyand in different locations around the intersection. Alternatively,instead of the sensor array 50 having a combination of cameras 52 anddetection and ranging sensors 54, the sensor array 50 may be limited toeither (i) an array of one or more cameras 52 oriented at differentangles and different directions, or (ii) an array of one or moredetection and ranging sensors 54 oriented at different angles anddifferent directions. In this alternative configuration, camera array 50and ranging sensor array 50 are used to distinguish between sensorsarrays having only one type of sensor.

The camera 52 may be a normal optical device relying on natural light tocapture images. The camera 52 may be configured to capture individualimages or a video stream. For example, the camera 52 may be configuredto capture sequential images or real-time video of road users at apredefined interval or frame rate with the captured images/video beingused by the GPU 22 to determine spatial-temporal data for each roaduser.

Images and videos captured by the camera 52 may be further processed bythe GPU 22 with machine vision algorithms to identify and track all theroad users within the viewing range of the camera 52 (e.g., 200 meters).

The camera 52 may include additional enhancements to reduce the camera'sreliance on natural light. For example, the camera 52 may includeartificial lights and flashes to provide better image capturingcapabilities. The camera may also include advanced sensors such as acomplementary metal-oxide-semiconductor field-effect transistor (CMOS)sensor for better capturing images in poor or low lighting conditions.Such sensors may be combined with artificial light such as infraredlighting for low light imaging and night vision capabilities.Alternatively, the camera 52 may be a thermographic camera such as aninfrared camera or a thermal imaging camera for capturing images of theroad users by using the heat signatures of the road users.

While the RSU 20 may use machine vision algorithms on the image datacaptured by the camera 52 to identify and track road users around theintersection, sequential still images and video streams of road userscaptured by the camera 52 may be processed by the GPU 22 to generatespatial-temporal data for all road users around an intersection.Spatial-temporal data acquired by the camera 52 may include thetrajectory, path, direction, bearing, and azimuth for all the trackedroad users. For example, image data captured by the camera 52 may beused to identify the trajectories of the road users and the changes inthe trajectories of the road users. The GPU 22 may also use image andvideo data from the camera 52 to calculate speed and acceleration of thetracked road users, but this data may be better acquired by thedetection and ranging sensor 54. The spatial-temporal data can befurther processed by the GPU 22 with A algorithms to predict thetrajectories, paths, and spatial-temporal behaviors of the road usersaround the intersection.

In addition to tracking the movement of road users to generatespatial-temporal data of the road users, the camera 52 may be used tocapture other data around the intersection. For example, the camera 52may be used to monitor the road condition and detect objects in the roadsuch as pedestrians, cyclists, animals, potholes, roadkill, lost loads,refuse, and the like, all of which may cause road users to swerve orbrake to avoid the object. That is, the camera 52 may correlate thedetected object to the trajectories, speeds, and accelerations of theroad users to develop spatial-temporal behavior patterns for the roadusers. For example, if road users in the road are swerving to avoid apothole, the GPU 22 may correlate the pothole to changes in the roadusers' trajectories when determining the predicted trajectories of theroad users. Such data can be used by the GPU 22 and applied to the AIalgorithms to better predict the trajectories and paths of the roadusers in view of such objects.

Likewise, the camera 52 may be used to monitor weather conditions todetermine if the weather may affect the behavior of the road users. Forexample, rain and snow may affect the road surface causing a moreslippery road surface and requiring extra time for road users to slow toa stop or necessitating extra care in driving/riding/walking on suchsurface. Such information can be used by the GPU to detect changes inthe road users' trajectories, speeds, and accelerations to identifyspatial-temporal behaviors. The GPU 22 may correlate such weatherconditions to the trajectory, speed, and acceleration data acquired bythe sensor array 50 and factor these spatial-temporal behaviors into thetrajectory and path predictions by the GPU 22. That is, the weather dataacquired by the camera 52 can be used by the GPU 22 and applied to the Aalgorithms to better predict the trajectories and paths of the roadusers in view of such weather conditions.

The sensor array 50 may also include one or more detection and rangingsensors 54. The detection and ranging sensor 54 may be configured tooutput a radio wave, receive the reflected radio wave, and measure atime from outputting the radio wave to receiving the reflected radiowave. The time measurement from the sensor 54 can be used as a basis fordetecting a road user and calculating the speed and acceleration of theroad user. For example, the ranging sensor 54 may output a radio wavetoward a road user and receive the radio wave reflected from the roaduser to detect and measure the speed and acceleration of the road user.As such, the detection and ranging sensor 54 may be a radar sensor 54.The detection and ranging sensor 54 may also be configured to output alight, such as infrared laser light, receive the reflected light, andmeasure a time from outputting the light to receiving the reflectedlight. By measuring a time to receive the reflected light, the detectionand ranging sensor 54 can use the time measurement as the basis fordetecting a road user and measuring the speed and acceleration of theroad user. As such, the detection and ranging sensor 54 may be a lidarsensor. The sensor array 50 may include one or more lidar and radarsensors 54 or a combination of lidar and radar sensors 54. The speedsand accelerations detected by the detection and ranging sensor 54 may beused by the GPU 22 to calculate spatial-temporal data andspatial-temporal behaviors for all the road users around theintersection. The spatial-temporal data/behaviors for all the road userscan then be further processed by the GPU 22 using A algorithms topredict the trajectories, paths, and spatial-temporal behaviors of theroad users around the intersection.

The sensor array 50, or individual cameras 52 and detection and rangingsensors 54, may be statically mounted at intersections to acquire a 360°view around the intersection. For example, at a four-way intersection, asensor array 50 or individual sensors 52 and 54 may be installed toacquire data for each of the four junction roads approaching theintersection (i.e., each junction road having a dedicated sensor array50). In this example, each sensor array 50 or sensor 52, 54 may beconfigured to have a 90° or greater field of view for each of thejunction roads approaching the intersection. Additional sensors arrays50 or individual sensors 52 and 54 may be installed to provide a 360°view within the intersection itself.

Alternatively, with reference to FIG. 2, the sensor array may use a panand tilt (PT) device 55 to rotate and tilt the sensor array 50 toward aroad user to capture data from the road user. The PT device 55 may bewired to the RSU 20 or communicate wirelessly with the RSU 20. The PTdevice 55 may include a controller 56 and one or more motors/actuators57 to control the rotation and tilt of the PT device 55. While FIG. 2shows a PT device 55 with the sensor array 50, individual sensors 52, 54may be used with the PT device 55 to rotate and tilt the sensors towardroad users to capture data. That is, the PT device 55 may be used to aimthe sensor array 50 at road users to capture spatial-temporal data fromthe targeted road users. More specifically, the apertures or lenses thatserve as the inputs/outputs of the sensors 52 and 54 may be aimed atroad users by the rotation and tilt capabilities of the PT device 55 tobetter capture the spatial-temporal data from road users. Motiondetection sensors around the intersection (not shown) may also be usedwith the PT device 55 to better target road users.

Actuators that may be employed by the PT device 55 include one or moreelectronic, electro-magnetic, motorized, linear screw, orpiezo-deformable actuators. For example, a linear screw actuator mayprovide fast, coarse targeting of the sensor array 50, while apiezo-deformable actuator can simultaneously provide a finer and moreprecise targeting. Motorized actuators may include servomotors andstepper motors.

The controller 56 of the PT device 55 may be used to calibrate the PTdevice 55 and return the PT device 55 to a “home” or reference position.Calibrating the PT device 55 may be performed algorithmically usingintrinsic and extrinsic parameters. The controller 56 is also configuredto receive targeting coordinates from the RSU 20, and in response,control the motors/actuators to rotate and tilt the sensor array 50based on the coordinates from the edge computing device 10.

The PT device 55 may also be used with one or more of the warningdevices 60 to control the targeting and targeted output of the warningdevices 60. For example, the PT device 55 may be used with a holosonicdevice 62 or a focused light display 64 to rotate and tilt the device 62or the light display 64 to target the output of the device 62 or thelight display 64. In other words, based on the risk assessmentscalculated by the edge computing device 10, the edge computing device 10can output the current and predicted paths of VRUs identified by therisk analysis as at-risk as control instructions to the PT devices 55used to rotate and tilt the device 62 and the light display 64. Thecontrollers 56 can use the current and predicted paths of the at-riskVRUs to aim the outputs of the holosonic device 62 or light display 64at the at-risk VRUs to provide a targeted warning at the precise,real-time locations of the at-risk VRUs.

The PT device 55 may be integrated with any of the sensor array 50, thecamera 52, the detection and ranging sensor 54, the holosonic device 62,and the focused light display 64 so that any of these components and thePT device 55 appear as one assembly. By using such an example integratedassembly, “targeting the holosonic device 62” may mean outputtingtargeting instructions to the controller 56 of the pan/tilt portion ofthe holosonic device 62 to rotate and tilt the device 62 to aim theoutput of the device 62 at a VRU.

With reference to FIG. 4, the sensor array 50 or individual sensors 52and 54 may be housed within a housing 58. The housing 58 may protect thesensor array 50 or individual sensors 52 and 54 from exposure to theelements. The housing 58 is sized appropriately to fit the sensor array50 or individual sensors 52 and 54, including the PT device 55.

With reference again to FIG. 1, the intersection infrastructure may useone or more warning devices 60 to warn VRUs of risk and danger scenariosand mitigate collisions between the VRU and a vehicle. The warningdevices 60 may be one or more holosonic devices 62, one or more focusedlight displays 64, and one or more drones 70, or any numberedcombination of holosonic devices 62, light displays 64, and drones 70.

The warning devices 60 are configured to receive targeting and outputinstructions from the RSU 20. The targeting and output instructions fromthe RSU 20 are based on the predicted trajectories and paths of the roadusers made by the GPU 22 by applying AI algorithms to thespatial-temporal data of the road users acquired by the sensor arrays 50and calculated by the GPU 22. The GPU 22 then further processes thespatial-temporal data of the road users and analyzes the data for risksto VRUs around the intersection to identify at-risks VRUs. At-risk VRUsidentified by the RSU 20 are tracked and targeted using both the sensordata from the sensor array 50 and the predicted trajectories of theat-risk VRUs computed by the GPU 22 using the correspondingspatial-temporal data. More specifically, the GPU 22 of the RSU 20 mayuse detection algorithms to precisely identify and localize the at-riskVRUs (i.e., targeted VRUs). The RSU 20 can then control the warningdevice 60 to output a targeted warning to the targeted VRU to warn thetargeted VRU of risk and danger. The warning device 60 may continue tooutput a targeted warning to the targeted VRU until the targeted VRUmoves away from the danger/risk scenario, or the danger/risk scenario nolonger poses a risk to the targeted VRU (i.e., the danger has passed).

The holosonic device 62 may be a holosonic speaker 62 and use modulatedultrasound to output a tight, narrow beam of targeted sound to a VRU.The holosonic device 62 may output an audible warning to a targeted VRUsuch as “stop,” “stay back,” “move back,” “do not walk,” “oncomingtraffic,” “danger,” alarm sound, siren, horn, or with another warningmessage or sound conveying danger to the targeted VRU. The holosonicspeaker 62 may also output sound at different volumes and intensitylevels commensurate to the danger. For example, the holosonic speaker 62may output a very loud targeted sound at a high volume to warn a VRU ofvery dangerous situations. The targeted sound output from the holosonicspeaker 62 to a VRU may only be audible or discernible by the targetedVRU. In this way, other road users not in danger may not hear thetargeted audio warning. As such, the other, not-at-risk road users maynot be caused to erroneously react to a warning message that is notintended for the road users. Such targeted warning messages may limitand/or prevent the not-at-risk road users from losing trust in thewarnings output by the intersection infrastructure 1.

The focused light display 64 may use focused light or imaging to outputa targeted warning light or image to an at-risk VRU (i.e., a targetedVRU). The targeted warning light or warning image may be only visible tothe targeted VRU. The focused light display 64 may use lights such aslasers and light emitting diodes (LEDs) to output the warning light orimage. The focused light display 64 may also use a display screen suchas an LED display, a quantum dot display (e.g., QLED), or like displayto output a targeted high intensity image, animation, or video to atargeted VRU. The targeted warning light may use a blinking or strobeeffect to get the attention of the targeted VRU to warn of danger. Forexample, the focused light display 64 may flash a focused, narrow-beamhigh intensity light in the face of a targeted VRU to get the attentionof the targeted VRU and warn the targeted VRU of danger. The focusedlight display 64 may also output light in a known warning color such asred, or output different light colors in a known display pattern such asthe red-yellow-green pattern of a traffic signal to instruct thetargeted VRU to start or stop moving. Similarly, videos and animationsoutput by the light display 64 may use known warning colors or visualeffects to better draw the targeted VRU's attention to the display.

The focused light display 64 may also output an instruction or symbol todenote or indicate danger such as “stop,” “stay back,” “move back,” “donot walk,” “oncoming traffic,” “danger,” the “Don't Walk” symbol or textused by pedestrian control signals, a car symbol, an exclamation pointinside a triangle, or like text and symbols denoting danger.

The light display 64 can also output specific instructions or a seriesof instructions to the targeted VRU in text or symbolic form to instructthe VRU on how to avoid the dangerous situation. For example, if theedge computing device 10 predicts a vehicle skidding off an icy road andonto a sidewalk with VRUs, the light display 64 may display arrowsindicating the directions the VRUs should move to avoid the skiddingcar. Likewise, the light display 64 may flash a directional arrow with adistance measurement indicating a VRU should move in the direction ofthe arrow by the given measurement to avoid danger.

The focused light display 64 can also be a holographic projector thatoutputs a targeted holographic image of any of the aforementioned text,symbols, animations, or videos in holographic form to warn the targetedVRU of danger.

The intersection infrastructure may use the holosonic device 62 andfocused light display 64 either individually, or in combination, tooutput both targeted audible and visible warnings to a targeted VRU.Similar to the sensor array 50, the holosonic speaker 62 and the focusedlight display 64 may be integrated into a single assembly such that theassembly can output both targeted audio and visual warnings.Alternatively, a plurality of holosonic speakers 62 can be arranged asits own array without any focused light displays 64. Likewise, aplurality of focused light displays 64 can be arranged as its own arraywithout any speakers 62.

The holosonic device 62 and the focused light display 64 may each use aPT device 55 shown in FIG. 2 to better output targeted warnings totargeted VRUs. Accordingly, warning devices 60 using PT devices 55 mayreceive targeting instructions from the RSU 20 to rotate and tilt thewarning devices 60 to precisely target the output of the warning device60 to the targeted VRU.

The intersection infrastructure warning system 1 may also use one ormore drones 70 to warn VRUs of potential danger. With reference to FIG.3, a schematic diagram of a drone 70 is shown with a nest 90.

The drone 70 may be an off-the-shelf commercial drone that is modifiedfor use in the intersection infrastructure warning system 1. Forexample, the drone 70 may be an off-the-shelf quadcopter with fourrotors. The drone 70 includes a controller 71, one or more motors 75,one or more rotors/propellers 76, a battery 77, an inductive coupling78, a GPS receiver 79, flight sensors 80, a camera 81, an ultrasonicsensor 82, a display 83, and a speaker 85.

The controller 71 may be a microcontroller or system on chip (SoC)configured for wireless communication. For example, the controller 71may be configured to transmit and receive encrypted wireless radiosignals using a wireless communication standard such as IEEE 802.11. Thetransmitting and receiving hardware of the controller 71 is not shown.The controller includes one or more processors 72 and memory 73. Thememory 73 may be a memory such as RAM, ROM, and flash memory, and thememory can be used to store programs and instruction sets that areexecutable by the processor 72 for controlling the flight of the drone.The controller 71 also includes an antenna 74 for transmitting andreceiving radio signals. For example, the controller 71 may receiveinstructions sets from the communication module 30 of the RSU 20 via theantenna 74. Instruction sets from the edge computing device 10 can bestored in memory 73 and executed by the processor 72. The controller 71may also control the power output to other electronic elements in thedrone 70, receive and process sensor data from sensing elements in thedrone 70, and output control signals to elements in the drone 70.

The controller 71 can control one or more motors 75 to control theflight of the drone 70. Each motor has a corresponding rotor/propeller76. For example, a quadcopter drone 70 may have four motors 75 and fourrotors 76. In lieu of a motor 75 and rotor/propeller 76 combination, thedrone 70 may use another propulsion system for flying the drone.

By controlling each of the motors 75 to control the rotation of therotors 76, the controller 71 can fly the drone 70 in any directiondepending on the control signals output to the motors 75. Thecommunication module 30 of the RSU 20 may output flight instructions(i.e., targeting instructions) directing the drone 70 to fly to atargeted VRU to warn the targeted VRU of risk or danger. Based on thetargeting instructions from the RSU 20, the controller 71 then controlsthe motors 75 to fly the drone to the targeted VRU. The instructionsfrom the RSU 20 may also include return instructions for directing thedrone 90 to return to the nest 90. In some instances, the returninstructions may be following a reverse flight path from the flight pathused to reach the targeted VRU.

The drone 70 may include one or more batteries 77 as a power source. Thebattery 77 may be rechargeable. For example, the battery 77 may bechargeable by an inductive coupling 78 for inductive charging of thebattery 77. The drone 70 may nest on a nest 90 positioned around anintersection that supports the drone 70 when the drone 70 is not inflight. That is, the nest 90 may act as a landing location and storageposition (i.e., “rest position”) for the drone 70 when the drone 70 isnot in use as a warning device 60. The nest 90 may also include aninductive coupling 92 that aligns with the inductive coupling 78 on thedrone 70 to charge the one or more batteries 77 of the drone 70 when thedrone 70 is not in use. The inductive coupling 92 is configured toreceive power from the electric infrastructure already present at theintersection for powering the other electric devices around theintersection.

The drone 70 may also include components such as the GPS receiver 79 andthe flight sensors 80 to properly direct and position the drone inflight. The GPS receiver 79 may be used by the controller 71 inconjunction with flight/targeting instructions from the RSU 20 tocontrol the motors 75 to ensure the drone 70 flies according to theinstructions from the RSU 20. The flight sensors 80 may include varioussensors such as multi-axis accelerometers and gyroscopes to ensure thatthe drone 70 stays level in flight with the correct position andorientation (e.g., right-side-up, oriented toward a targeted VRU).Accordingly, the controller 71 may use such data from the flight sensors80 to control the motors to fly the drone with the correct position andorientation.

The drone 70 may also include a camera 81 and/or an ultrasonic sensor82. The camera 81 and the ultrasonic sensor 82 may be disposed on theunderside (i.e., bottom) of the drone 70 and used to help land thedrone. For example, image data from the camera 81 may be used by thecontroller 71 to control the motors 75 to land the drone 70 with thecorrect position and orientation on the nest 90 (i.e., to properly alignthe inductive coupling 78 on the drone 70 with the inductive coupling 92on the nest 90). Alternatively, or in addition to the camera 81, anultrasonic sensor 82 may be used to help land the drone 70 on the nest90. The ultrasonic sensor 82 may output ultrasonic waves and sensereflected waves to determine the distance from an object (e.g., the nest90). Sensor data from the ultrasonic sensor 82 can be used by thecontroller 71 to control the motors 75 to properly land the drone 70 onthe nest 90. The ultrasonic sensor 82 may also be used when the drone 70is hovering to maintain the drone at a fixed height relative to theground during hovering. In place of the ultrasonic sensor 82, aprecision altimeter or other sensor may be used for landing the droneand maintaining a fixed height during a hovering operation.

Alternatively, when the drone 70 is used to direct a targeted VRU tosafety, the camera 81 and the ultrasonic sensor 82 may be used to sensethe position and motion of the targeted VRU to ensure that the drone 70does not contact the targeted VRU, and that the targeted VRU is movingaway from the risk scenario. The camera 81 and the ultrasonic sensor 82may also be used to identify other objects in the instructed flight pathof the drone 70 so that the drone 70 may avoid such objects in theflight. Alternatively, or in addition to using the camera 81 and theultrasonic sensor 82, the GPS receiver 79 and the flight sensors 80 maybe used while the drone 70 is in flight for collision avoidance.

The drone 70 may also include a display 83 and/or a speaker 85 foroutputting additional visual and audible warnings and instructions to atargeted VRU.

The display 83 may be a high tech or low tech display. For example, thedisplay 83 may be a printed banner or sign with a warning message,symbol, or instructions. Alternatively, the display 83 may be anelectronic display that can display a video message, symbol,instruction, animation, or video that warns a targeted VRU of danger.The display 83 may include a display motor/actuator 84 that can positionthe display for appropriate viewing by a targeted VRU. For example, thedisplay 83 may be positioned in a first position when the drone 70 is inflight for better aerodynamics while flying. The controller 71 may thencontrol the display motor 84 to position the display 83 in a secondposition for best viewing by the targeted VRU as the drone 70 approachesthe targeted VRU. In other words, the display motor 84 can change theposition of the display 83 to better fly and land the drone, and thenreposition the display 83 for better visibility by the targeted VRU. Theinstructions from the RSU 20 may include instructions for changing theposition of the display. For example, the instructions may indicate itwill take x amount of time for the drone 70 to reach the targeted VRU.At x−1 time, the instructions may indicate controlling the display motor84 to change the position of the display 83 from the first position tothe second position for viewing by the VRU. At x+y time, theinstructions may indicate controlling the display motor 84 to change theposition of the display 83 from the second position to the firstposition for the return flight to the nest 90. In this example, theexample times may be measured in seconds and x and y may be positivenatural numbers or some positive fractional component thereof.

Messages displayed by the focused light display 64 and by the display 83of the drone 70 should be short and direct to evoke a rapid response bythe VRU.

While the holosonic speaker 62, the focused light display 64, and thedrone 70 were described as warning devices 60, the warning devices ofthe intersection infrastructure warning system are not limited to thesewarning devices 60. Other warning systems providing different sensorystimulation to a VRU may deployed. For example, the warning device 60may output a stream or burst of jetted fluid such as air or water. Assuch, the warning device 60 may be a jetted fluid output device. Inextreme situations where a VRU may be sensory impaired, for example, bywearing headphones and listening to loud music, the sensory impairmentof the VRU may be identified by the edge computing device 10 so as tooutput a warning commensurate to the VRU's impairment. In someinstances, a jetted fluid output device may be used to output a jettedstream of fluid get the attention of a VRU by affecting non-impairedsenses of the VRU. While a VRU may be annoyed with being hit by an airor water stream from a jetted fluid output warning device, suchannoyance may be better than a collision with a vehicle.

With reference to FIG. 4, an example intersection 100 where theinfrastructure warning device 1 can be used is illustrated. Theintersection 100 includes traffic control signals 102, pedestriancontrol signals 104, marked pedestrian crossing points (e.g.,crosswalks) 106, and stop lines 108 for controlling the flow ofpedestrians 120, cyclists 122, and motor vehicles 130 (collectivelyreferred to as “road users”). The intersection 100 also includes theelements of the intersection infrastructure warning system 1 includingthe RSU 20 (not shown) housed in the traffic cabinet 21 mounted on autility pole 103, the sensor arrays 50 (not shown) in housings 58, andthe drones 70. In the example intersection warning infrastructure 1shown in FIG. 4, the drones 70 are used as the warning device 60. Thenests 90 for the drones 70 may include individual support poles 94.However, the nests 90 may also be installed on existing intersectioninfrastructure like utility pole 103.

While a four-way intersection is shown in FIG. 4, the infrastructurewarning system 1 may also be used at more complex intersections with agreater number of junction roads, at roundabouts, and at intersectionswith less junction roads (e.g., three-way intersections).

The RSU 20 can determine the status of the traffic control signals 102and the pedestrian control signals 104 through a wired or wirelessconnection. That is, the RSU 20 is configured to receive SPaT messagesfrom the control signals 102 and 104 to determine the status of thecontrol signals 102 and 104. Alternatively, the RSU 20 may determine thestatus of the control signals 102 and 104 by the cameras 52.

The image and motion data acquired by the sensor arrays 50 is used bythe RSU 20 with AI algorithms to determine the predicted trajectoriesand paths of the road users 120, 122, and 130. That is, the sensorarrays 50 collect data to detect, localize, and track all road users120, 122, and 130 in and around the intersection 100. The RSU 20 may useimage processing and machine vision to identify and track the road users120, 122, and 130. As shown in FIG. 4, the RSU 20 may superimpose boxes140 on captured video from the cameras 52 to identify and track the roadusers 120, 122, and 130 around the intersection 100. The RSU 20 can thenuse the detection and ranging sensors 54 to acquire measurements fordetermining the spatial-temporal data of the road users 120, 122, and130 such as trajectory, path, direction, speed, and acceleration

The data acquired by the sensor arrays 50 can be used by the RSU 20 tocompute proxy basic safety messages (BSMs) from the motor vehicles 130and to compute proxy personal safety messages (PSMs) for the pedestrians120 and cyclists 122 (i.e., the vulnerable road users (VRUs)). That is,the RSU 20 can compute proxy spatial-temporal data for the road users120, 122, and 130 in lieu sensors on the road users gathering thespatial-temporal data. The RSU 20 can then use the proxyspatial-temporal data with AI algorithms to predict the trajectories,paths, intent, and behavior patterns of the road users 120, 122, and130.

The proxy BSMs calculated by the RSU 20 may include a subject vehicle'sspeed and acceleration in addition to the subject vehicle's distance tothe stop line 108, the distance from the subject vehicle to a leadvehicle (i.e., a vehicle traveling in front of the subject vehicle), thevelocity and acceleration of the lead vehicle, the heading or steeringangle of the subject vehicle, and the status of the traffic controlsignal 102. The proxy BSMs for the vehicles 130 can be used with AIalgorithms by the GPU 22 to predict the trajectories and drivingbehavior of the motor vehicles 130. Alternatively, vehicles havingdedicated short-range communications (DRSC) and connected tovehicle-to-everything (V2X) systems may transmit CAN data from vehiclesensors including velocity, throttle opening rate, brake pressure, andsteering angle to communicate a BSM to the RSU 20 via the communicationmodule 30. The CAN data from connected vehicles can be used with AIalgorithms in addition to the proxy BSMs calculated by the RSU 20 topredict the trajectories and driving behavior of the connected vehicles130.

The proxy PSMs calculated by the RSU 20 may include the speed,acceleration, and travel direction of the pedestrians 120 and cyclists122 in addition to speed and acceleration data for other pedestrians andcyclists around the subject pedestrians and cyclists, and the status oftraffic control signals 102 and pedestrian control signals 104. Theproxy PSMs for the pedestrians 120 and cyclists 122 can be used by theGPU 22 with A algorithms to predict the paths, trajectories, intent, andbehavior of the pedestrians 120 and the cyclists 122. Alternatively, forpedestrians 120 and cyclists 122 that are connected to the V2X systemvia a cell phone or other personal communication device, data acquiredfrom the phone or other personal communication device related to thepedestrian's 120 and cyclist's 122 speed, acceleration, and traveldirection can be used to communicate a PSM of the pedestrian 120 orcyclist 122 to the RSU 20 via the communication module 130. The speed,acceleration, and travel direction data from the connected pedestrian120 or cyclist 122 can be used with AI algorithms in addition to theproxy PSMs calculated by the RSU 20 to predict trajectories, paths, andspatial-temporal behaviors of the connected pedestrians 120 or cyclists122. The other personal communication device could be a wearablecommunications or computing device using ultra wide band (UWB)technology with a unified communication and location system that allowsfor the localization of the other personal communication device with anaccuracy of within ten centimeters or less. The other personalcommunication device with UWB technology may also be enabled for highspeed communication measured in microseconds or nanoseconds. The highspeed communication and more precise localization of a personalcommunication device using UWB may used for quickly acquiring data fromthe pedestrians 120 and the cyclists 122 for calculating a proxy PSM.

The predicted trajectories, paths, and behaviors of the road users 120,122, and 130 are derived for the VRUs 120 and 122 using thespatial-temporal data in the proxy PSM and broadcast PSM data, andderived for the vehicles 130 using the spatial-temporal data in theproxy BSM and broadcast BSM data. The GPU 22 then uses an AI algorithmto further process the spatial-temporal data for each respective PSM/BSMto derive behavior intent and predict speed, trajectory, path, headingwith an extended time horizon greater than three second (i.e., >3 sec).The GPU 22 then uses an AI algorithm to process the VRU data for theVRUs 120 and 122 and the vehicle data for the vehicles 130 to localizewhich of the vehicles 130 around the intersection 100 pose threats tothe VRUs 120 and 122 around the intersection 100.

For connected vehicles 130, that is, vehicles 130 having DSRC andconnected to the V2X system, a unique vehicle ID may be used by the Aalgorithm executed by the GPU 22 to identify which vehicles 130 poserisks to the VRUs 120 and 122, and then the GPU 22 correlates such riskto the at-risk VRUs 120 and 122. In such manner, the RSU 20 may onlyoutput alert messages to the connected vehicles 130 posing a risk to theVRUs (i.e., targeted vehicles 130) so that other connected vehicles 130around the intersection 100 (i.e., non-risk or non-targeted vehicles130) are not distracted by the alert message. The alert message outputby the RSU 20 to the targeted vehicles 130 may cause an alert or displaymessage to appear on the instrument panel, infotainment display, orhead-up display of the targeted vehicle 130. The alert message may alsocause a warning light or directional arrow pointing in the direction ofthe at-risk VRU to appear on a side view mirror or A-pillar of thetargeted vehicle 130 to alert the driver to the possibility of acollision with the at-risk VRUs 120 and 122. Alternatively, or inaddition to the visual warning in the connected vehicle 130, the alertmessage output by the RSU 20 may cause the speakers of the vehicle'sinfotainment device to output speech or a warning sound to alert thedriver of the vehicle 130 to the possibility of a collision with theat-risk VRUs 120 and 122.

The GPU 22 may assign some other identification to non-connectedvehicles 130 when the proxy BSM for such vehicles is processed andanalyzed by the edge computing device 10. However, the RSU 20 of theintersection infrastructure warning system 1 may not be able to outputtargeted, in-vehicle alert messages to non-connected vehicles 130 (i.e.,vehicles 130 without DSRC and not connected to V2X systems).

The predicted trajectories and paths of the road users 120, 122, and 130can be further analyzed, for example, by comparing these trajectoriesand paths to trained models based on past data acquired by the sensorarray 50 and calculated by the cloud 40. The trained models calculatedby the cloud 40 are periodically updated using the data acquired by thesensor array 50.

After the RSU 20 identifies the vehicles 130 around the intersection 100that pose a risk to the VRUs 120 and 122 at the intersection 100, theRSU 20 can output targeting instructions to the warning devices 60 towarn the at-risk VRUs 120 and 122 (i.e., the targeted VRUs 120 and 122).In this example embodiment shown in FIG. 4, the RSU 20 outputs flightinstructions to one or more drones 70 at the intersection 100 to deploythe drones 70 and warn the targeted VRUs 120 and 122.

A process flow 500 for the processes performed by the edge computingdevice 10 is described with reference to the flowchart in FIG. 5.

At S502, the edge computing device 10 uses machine vision algorithms toprocess image data from the cameras to detect and track all road usersaround an intersection. The edge computing device 10 may also usemeasurements from the detection and ranging sensor 54 to detect roadusers around the intersection. The edge computing device 10 continues totrack all the road users around the intersection so long as the roadusers are in range of the sensor array 50 (i.e., approximately 200meters).

At S504, the edge computing device 10 continuously acquires data fromthe sensors 52 and 54 as the basis for calculating spatial-temporal datafor each road user such as trajectory, path, direction, bearing,heading, azimuth, speed, and acceleration, in addition to identifyingspatial-temporal behavior for all the tracked road users.

At S504, the edge computing device 10 may also acquire BSM and PSMmessages from tracked road users connected to the intersectioninfrastructure warning system 1 via V2X devices and systems. The GPU 22may use the BSM and PSM messages as the basis for calculating thespatial-temporal data for the connected road users in addition to usingthe respective data of the connected road users acquired by the sensors52 and 54.

At S506, the GPU 22 of the edge computing device 10 processes thespatial-temporal data acquired at S504 for all the tracked road users tocalculate the current paths, trajectories, directions, bearings,headings, azimuths, speeds, and accelerations for all the tracked roadusers, in addition to identifying the spatial-temporal behaviors for allthe tracked road users.

At S508, the GPU 22 further processes the calculated spatial-temporaldata and spatial-temporal behaviors for all the tracked road users usingAI algorithms to predict the paths, trajectories, intents, andspatial-temporal behaviors for all the tracked road users.

At S510, the GPU 22 further analyzes the calculated spatial-temporaldata and identified behavior for all the tracked road users with thepredicted paths, trajectories, intents, and spatial-temporal behaviorsof all the road users using AI algorithms along multiple trajectories.That is, the edge computing device 10 may calculate predictions for eachof the road users along multiple trajectories. The multiple trajectoryanalysis for each of the road users is further analyzed for predictedconvergences between the road users that pose imminent threatinteractions (e.g., collisions with a VRU) to identify conflict zones.

At S512, if the GPU 22 does not identify any conflict zones, the processreturns to S502 and repeats. If the GPU 22 identifies conflict zones,i.e., “YES” at S512, the process proceeds to S514.

At S514, the edge computing device 10 determines and prioritizesimminent threats. The determination process includes identifying theVRUs in conflict zones, the threats to the VRUs in the conflict zones,and the time horizons of the threats to the VRUs in the conflict zones.For example, the GPU 22 may identify a particular VRU, the threat to theVRU, and the time until the threat occurs to the VRU. The GPU 22 thenranks all the threats on the basis of the threat and the time horizon.For example, for a particular VRU, the GPU 22 may identify the threat isa side impact to the VRU with a seven second time horizon. The GPU 22may then classify this threat by assigning a threat level such as low,medium, or high, or some other identifier like a number. The edgecomputing device 10 may use a predetermined threat matrix, lookup table,or other index fetched from the storage 26 to more quickly correlate thethreat and time horizon with a threat level. The responses to threatsmay then be ranked and prioritized by the GPU 22 based on the threatlevel. For example, the edge computing device 10 may prioritize aresponse to a threat with a high threat level over a response to athreat with a lower threat level.

The threat level and predicted time horizons related to the threats mayfacilitate warning responses customized for each VRU 120 and thepredicted risk determined by the GPU 22 for each VRU 120. For example,the warning response may be to warn one or more VRUs 120 to move left,right, diagonally backward, or to turn around and run, to avoid danger.

At S516, the edge computing device 10 determines a response on the basisof the threat and threat level. That is, the edge computing device 10selects a warning device 60 and determines the output of the warningdevice based on the threat and the threat level. For example, the edgecomputing device 10 may determine at S514 that a side impact threat to aVRU with a seven second horizon has a low threat level. In response, theedge computing device may deploy a drone 70 to fly to the VRU head onand display a message “step back” on the display 83 of the drone 70.Like the threat prioritization, the edge computing device 10 may use apredetermined matrix, lookup table, or other index fetched from thestorage 26 to correlate the threat and threat level to a response, inlieu of the GPU 22 having to perform extra processing to determine thebest response.

With reference to FIG. 6, an example warning response matrix 600 isillustrated. A matrix, lookup table, or other index may be stored in thestorage 26 and used by the edge computing device 10 to quicklyprioritize threats and determine a response. The response matrix 600 mayinclude a threat ranking 602, a characterization of the threat to theVRU 604, a threat level 606, and a characterization of the response 608.

The threat ranking 602, threat 604, and threat level 606 are determinedby the edge computing device at S514. However, the warning responsematrix 600 may be used by the edge computing device 10 to more quicklycorrelate the identified threat with a threat level by storingpredetermined correlations in the form of a matrix or lookup table inthe storage 26. In this case, once the edge computing device 10determines the threat at S514, a matrix can be used to correlate thethreat 604 to a threat level 606 without the edge computing device 10having to perform extra processing to make a threat level determination.A plurality of threats 604 can then be ranked based on the threat level606.

The threats 604 can also be correlated to a response 608 so that theedge computing device 10 can use the warning response matrix 600 in lieuof having to determine a response. Such correlations can eliminateunnecessary processing by the edge computing device 10 to provide aneven faster real-time response. Such correlations also ensureconsistency in the response to the threats. Consistent responses mayfamiliarize the road users with the warnings from the intersectioninfrastructure warning system 1 and increase the users' trust in thewarnings and the warning system 1.

The edge computing device 10 may use the characterization of theresponse 608 as response instructions for the warning device 60 in lieuof having to having to perform extra processing to determine the bestresponse. That is, the edge computing device 10 may translate theresponse characterization 608 into machine instructions and output suchmachine instructions as targeting and response instructions to thewarning devices 60.

In the example shown in FIG. 6, the edge computing device 10 outputsresponse instructions 608 to one or more drones 70. However, warningresponse matrixes 600 may include additional threat characterizationsand response characterizations, including the deployment of differentwarning devices 60 to warn a VRU of danger. The degrees of freedom basedon the number of threat scenarios will determine the configuration ofany warning response matrix, lookup table, or other index.

Alternatively, data accumulated by the RSU 20 may be sent to the cloud40 for modeling and simulation. That is, the cloud 40 of the edgecomputing device 10 can develop trained models using a wide-range ofdifferent simulated threat and response scenarios. These models may beused by the edge computing device 10 to determine a warning responsematrix 600 or used in combination with actual threat and responsescenarios to develop the warning and response matrix 600.

With reference again to FIG. 5, at S518, the edge computing device 10determines targeting instructions for the warning device 60 used in theresponse determined by the edge computing device 10 at S516. The edgecomputing device 10 may use the calculated spatial-temporal data andcalculated predictions to precisely target the warning device 60 on theVRU. For example, based on the known spatial-temporal data of the VRUcalculated by the GPU 22 at S506 and the predicted path, behavior, andintent of the VRU determined by the GPU 22 at S508, the edge computingdevice 10 can determine the position of the VRU at any given time toensure that the warning device 60 is precisely targeted on the VRU.

For the holosonic speaker 62 and the focused light display 64 warningdevices 60 that may be used with a PT device 55, the targetinginstructions from the edge computing device 10 may include theinstructions to the controllers 56 to control the rotation and tilt ofthe PT device 55 to aim the output of the warning device 60 at the VRU.

For the drones 70, the targeting instructions from the edge computingdevice 10 may be flight instructions with the flight path the drone 70should fly to approach the VRU.

At S520, the edge computing device 10 outputs the targeting and responseinstructions to the warning devices 60 and the warning devices 60 warnthe VRU.

At S522, the edge computing device 10 determines whether the warningdevices 60 have mitigated the danger. That is, the edge computing device10 may determine whether the danger is still ongoing and/or maydetermine whether the warning devices 60 have been successful in warningthe VRU to mitigate the danger. If the edge computing device 10determines that the danger has not been ameliorated, i.e., “NO” at S522,the process proceeds to S524.

At S524 the edge device 10 may update and output revised targetingand/or response instructions, if necessary, until the edge devicedetermines that the danger has been mitigated at S522.

When the edge computing device 10 determines that the warning devices 60have mitigated the danger at S522, the process proceeds back to S502.The process 500 by the edge computing device 10 is a continuousmonitoring and response process.

The edge computing device 10 may output complete instructions at S520without the feedback processes of S522 and S524. That is, the edgecomputing device 10 may output targeting and response instructions tothe warning device 60 with additional instructions including timeinstructions for completing the warning response and return instructionsfor the warning device 60. For example, in determining the time horizonof the threat at S514, the edge computing device 10 may additionallydetermine that the threat to the VRU will pass after x amount of time(i.e., the VRU will no longer be in danger after x amount of time). AtS516, the response instructions to the warning device 60 may includeexecuting return instructions after the x amount of time has elapsed.That is, if the edge computing device 10 determines that a threat willbe over in ten seconds, the response instructions to the warning device60 may include instructions to begin a return process after the tenseconds have elapsed.

For the warning devices 60 using a PT device 55 to provide a targetedoutput to the VRU, return instructions may be stopping the output of thewarning device 60 and instructing the PT device 55 to return to a homeposition, as determined by the calibration the PT device 55.

For the drone 70, return instructions may include stopping the warningoutput, adjusting the position of the display 83 for flight, ifnecessary, and returning to the nest 90. The on-board sensors of thedrone 70 may be used for returning to and landing properly on the nest90, as described above with reference to FIG. 3.

Alternatively, when the edge computing device 10 and the warning devicesare configured for real-time updates and include the feedback processesat S522 and S524, the edge computing device 10 may include transmittingreturn instructions to the warning devices 60 in response to determiningthe danger has been mitigated at S522. That is, the edge computingdevice 10 may output return instructions to the warning devices 60before the process proceeds to S502 to repeat.

Because of the increased computational abilities of the edge computingdevice 10, the response and targeting instructions are made by the edgecomputing device 10 in lieu of processing by the controller 56 in the PTdevice 55 and the controller 71 in the drone 70. This allows theintersection infrastructure warning system 1 to use off-the-shelfdevices like the PT device 55 and the drone 70 while limitingmodifications to the controllers of these devices.

With reference to FIGS. 7-11, operational examples for exampleembodiments of the intersection infrastructure warning system 1 atintersection 100 are illustrated and described.

With reference to FIG. 7, the example intersection infrastructurewarning system 1 at intersection 100 includes warning an at-risk VRU 120by a drone 70 and further risk remediation by the drone 70. After theedge computing device 10 of the infrastructure warning system 1identifies a vehicle 130 posing a risk to a VRU 120, the edge computingdevice 10 outputs warning message format, delivery mode, and flightinstructions to the drone 70.

The drone 70 then deploys from the nest 90, drops vertically down intothe path of the VRU 120, and hovers directly in front of the VRU 120.The drone 70 displays a warning message and outputs an audio warning toalert the VRU 120 of danger from the vehicle 130. In the example shownin FIG. 7, the drone 70 outputs a visual warning to the VRU 120 on thedisplay 83 with “move back” text in addition to outputting an audiowarning 85 a from the speaker as a recorded message instructing the VRU120 to “move back.”

The drone 70 then maintains a position directly in front of the VRU 120and avoids contact with the VRU 120 to block forward movement of the VRU120 to prevent the VRU 120 from walking into danger—in this case, thepath of the vehicle 130. The drone 70 may then fly toward the VRU 120,without contacting the VRU 120, in an attempt to direct (i.e., push) theVRU 120 away from the dangerous location in order to remedy thedangerous situation. In this manner, the drone 70 can both warn atargeted VRU 120 and remedy a dangerous situation by directing thetargeted VRU 120 away from the danger.

With reference to FIG. 8, the example intersection infrastructurewarning system 1 at intersection 100 includes warning multiple at-riskVRUs 120 a and 120 b using multiple drones 70 a and 70 b with furtherrisk remediation by the drones 70 a and 70 b. For example, in thesituation shown in FIG. 8, the edge computing device 10 may predict theVRU 120 a as eager to cross the road in anticipation of the controlsignal 102 a changing from yellow to red, while also predicting that thevehicle 130 a will not stop or slow down despite the change of thecontrol signal 102 a from yellow to red (i.e., predicting the vehicle130 a will run a red light). In this case, the VRU 120 a may run intothe path of vehicle 130 a. Similarly, the edge computing device 10 mayanalyze the running VRU 120 b and predict the VRU 120 b will try tocross the intersection despite the control signal 102 a changing fromyellow to red. The edge computing device 10 may also predict that driverof the vehicle 130 b is anticipating the change of the control signal102 b from red to green and may avoid braking to slow to complete stop.In this case, the vehicle 130 b may not brake, or late brake, which theedge computing device 10 may identify as causing a predicted collisionwith the running VRU 120 b.

After the edge computing device 10 of the infrastructure warning system1 identifies the vehicle 130 a posing a risk to the VRU 120 a and thevehicle 130 b posing a risk to the VRU 120 b, the edge computing device10 outputs respective warning message formats, delivery modes, andflight instructions (i.e., response and targeting instructions) to thedrones 70 a and 70 b.

The drone 70 a then deploys from the nest 90 a located immediatelyoverhead of the VRU 120 a, similar to the flight of the drone describedwith reference to FIG. 7. That is, the drone 70 a may drop from nest 90a, fly toward, and hover in front of the VRU 120 a to block the path ofthe VRU 120 a. The drone 70 a may display a “stop” warning message onthe display 83 a and output an audio warning 85 a from the speakerinstructing the VRU 120 a to “stop.”

The drone 70 a continues to hover to maintain a position directly infront of the VRU 120 a and avoids contact with the VRU 120 a to blockthe forward movement of the VRU 120 a and prevent the VRU 120 a fromwalking into danger—in this case, the path of the vehicle 130 a. Thedrone 70 a may then fly toward the VRU 120 a, without contacting the VRU120 a, in an attempt to direct (i.e., push) the VRU 120 a away from thedanger to remedy the dangerous situation.

Similarly, the drone 70 b may deploy from the nest 90 b, fly across thestreet, and drop (i.e., fly downward) to block the path of the VRU 120b. The drone 70 b may display a “move back” warning message on thedisplay 83 b and output an audio warning 85 b from the speakerinstructing the VRU 120 b to “move back.”

The drone 70 b then continues to hover to maintain a position directlyin front of the VRU 120 b, avoiding contact with the VRU 120 b, to blockforward movement of the VRU 120 b and prevent the VRU 120 b from walkinginto danger—in this case, the path of the vehicle 130 b. The drone 70 bmay then fly toward the VRU 120 b, without contacting the VRU 120 b, inan attempt to direct (i.e., push) the VRU 120 b away from the danger toremedy the dangerous situation.

In this manner, the drones 70 a and 70 b can respectively warn thetargeted VRUs 120 a and 120 b and also remedy dangerous situations bydirecting the targeted VRUs 120 a and 120 b away from the danger.

While the example shown in FIG. 8 shows two drones 70 a and 70 bdeploying from two nests 90 a and 90 b, any number of drones 70 andnests 90 may be used. For example, the drones 70 a and 70 b may bothdeploy from a single nest (e.g., the nest 90 a). In another example, theintersection 100 may include a nest 90 on each corner of theintersection 100 with one or more drones 70 deploying from each nest. Inthis manner, the drones 70 may be able to immediately drop from overheadnests 90 to more quickly alert and block the forward travel of a VRU120.

With reference to FIG. 9, the example intersection infrastructurewarning system 1 at intersection 100 includes warning an at-risk VRU 120using multiple drones 70 a and 70 b with further risk remediation by thedrones 70 a and 70 b. For example, the VRU 120 at intersection 100 maybe heavily distracted by the display of a smart phone device withfurther sensory impairment from the audio output of the smart phonedevice via headphones. In such an example, the hearing of the VRU 120may be impaired by the headphones such that the VRU 120 may not be ableto perceive audio warnings from the vehicle 130 or the drone 70 a, whilethe user is otherwise engaged in the display of the smart phone deviceand oblivious to his surroundings. The sensory impairment of the VRU 120may be so great that the VRU 120 may not realize he is walking intodanger despite warnings from one drone 70 a. In this example, the edgecomputing device 10 may predict, based on the spatial-temporal data andbehavior of the VRU acquired by the sensors, that the VRU 120 willcontinue walking into the intersection 100 without heeding thepedestrian control signals. In another example, based on the threat andwarning matrix in FIG. 6, the edge computing device 10 may determinethat the vehicle 130 is exceeding the speed limit and predict a sidecollision between the VRU 120 and the vehicle 130 with only a threesecond horizon for warning the VRU 120. In this example, the edgecomputing device 10 may deploy both the drones 70 a and 70 b to warn theVRU 120 of danger. In both examples, the edge computing device 10 maydeploy a second drone 70 b in conjunction with a first drone 70 a towarn the VRU 120 of danger and mitigate the dangerous situation.

After the edge computing device 10 of the infrastructure warning system1 identifies the vehicle 130 posing a risk to the VRU 120, the edgecomputing device 10 outputs a warning message format, a delivery mode,and flight instructions to at least the drone 70 a. The edge computingdevice 10 may simultaneously output a warning message format, a deliverymode, and flight instructions to the drone 70 b to deploy both drones 70a and 70 b simultaneously. Alternatively, after deploying the drone 70a, the edge computing device, based on the continuously acquired sensordata from the sensor array 50, may determine a change in threat levelbased on the predictions and determine that additional warnings shouldbe deployed. That is, the edge computing device 10 may instruct thedrone 70 b to deploy after first deploying the drone 70 a.

The drone 70 a deploys from the nest 90 a located immediately overheadof the VRU 120 a, similar to the flight of the drone described withreference to FIG. 7. That is, the drone 70 a may drop from nest 90 a,fly toward, and hover in front of the VRU 120 to block the walking pathof the VRU 120 a. The drone 70 a may display a “stop” warning message onthe display 83 a and output an audio warning 85 a from the speakerinstructing the VRU 120 to “stop.”

The drone 70 b either simultaneously deploys, or deploys after thedeployment of the drone 70 a, flies from the nest 90 b across the streetfrom the VRU 120, drops to the left side of the VRU 120, and hovers onthe side of the VRU 120 next to the drone 70 a to block the walking pathof the VRU 120. The drone 70 b may display a “move back” warning messageon the display 83 b and output an audio warning 85 b from the speakerinstructing the VRU 120 to “move back.”

The drones 70 a and 70 b then continue to hover to maintain side-by-sidepositions in the forward path of the VRU 120, while avoiding contactwith the VRU 120, to create a larger barrier for blocking the forwardmovement of the VRU 120. In this way, the drones 70 a and 70 b can beinstructed to swarm the VRU 120 and overwhelm the senses of the VRU 120to force the VRU 120 to heed the warnings of the drones 70 a and 70 b.Alternatively, or in addition to the swarm effect, the drones 70 a and70 b can also be used to create a larger wall effect and prevent the VRU120 from walking into danger—in this case, the path of the vehicle 130.The drones 70 a and 70 b may then fly toward the VRU 120 in acoordinated manner, without contacting the VRU 120, to direct the VRU120 away from the danger to remedy the dangerous situation.

While the example of FIG. 9 shows the drone 70 b flying from a nest 90 bacross the street from the VRU 120, other arrangements are possible. Forexample, the nest 90 a may be used to park both drones 70 a and 70 bsuch that both drones 70 a and 70 b deploy from the same nest. That is,multiple drones 70 may deploy from one nest 90. The infrastructurewarning system 1 may include nests 90 on each corner of the intersection100 where each nest 90 hosts one or more drones 70.

The example embodiments described with reference to FIGS. 9 and 10 bothuse the simultaneous deployment of drones 70 a and 70 b as a warningdevice 60. For simultaneous readiness, multiple drones 70 can bestrategically placed in nests 90 around an intersection 100 to form“hives” and launched as needed to multiple simultaneous targets. Here,the drones 70 can maintain a full power cell 77 charge, and thecontrollers 71 of the drones 70 are continually updated with essentialintersection road users' (e.g., 120, 122, 130) statuses and flightinformation along with coordinated cognitive “swarm” policies. That is,multiple drones can be prepared for real-time interventions for singleor multiple VRUs 120 to execute unique safety strategies customized tothe unique risks and behavior of individual VRUs 120.

With reference to FIG. 10, the example intersection infrastructurewarning system 1 at intersection 100 includes warning an at-risk VRU 120using other warning devices 60, such as one or more holosonic speakers62 and one or more focused light displays 64. The example intersectioninfrastructure warning system 1 in FIG. 10 warns the VRU 120 of dangerfrom the vehicle 130.

Both the holosonic speakers 62 and the focused light displays 64 may bemounted to the existing mounting structures used by the traffic controlsignals 102, such as the utility poles 103. In the example embodimentshown in FIG. 10, the speakers 62 and light displays 64 are installed atthe tops of the utility poles 103, but the installation position is notlimited to this position. The speakers 62 and light displays 64 may bedisplayed at other locations on the utility poles 103 or may haverespective individual utility poles intended only for mounting thespeakers 62 and/or the light displays 64, similar to the nest utilitypoles 94 shown in FIG. 4.

The holosonic speakers 62 and the focused light displays 64 can use thePT device 55 to better aim the outputs of the speakers 62 and lightdisplays 64 for better targeting the VRU 120 with focused sound andlight warnings. That is, the speakers 62 and the light displays 64 canbe mounted to PT devices 55 and the PT devices 55 can be installed onthe utility poles 103. Alternatively, the PT devices 55 may beintegrated with the speakers 62 and the light displays 64.

The intersection infrastructure warning system 1 may use one or moreholosonic speakers 62 in place of drones to warn the VRU 120 of thevehicle 130. After the edge computing device 10 of the infrastructurewarning system 1 identifies the vehicle 130 posing a risk to the VRU120, the edge computing device 10 outputs an audio warning format andtargeting instructions to the holosonic speaker 62. The PT device 55 (orthe pan/tilt portion of an integrated speaker design) then rotates andtilts the speaker 62 to aim the output of the speaker 62 toward the VRU120. The holosonic speaker 62 then outputs a holosonic audio warning 63to the targeted VRU 120. For example, the holosonic audio warning 63 maybe an audio message instructing the VRU 120 to “stop.” The holosonicaudio warning 63 may be audible only at a specific location. That is,the holosonic audio warning 63 may only be audible immediately aroundthe position of the VRU 120. In this manner, the intersectioninfrastructure warning system 1 can warn only the targeted VRU 120 whileother pedestrians at intersection 100 do not hear the warning 63.Consequently, the other pedestrians are not erroneously warned ofdanger, which can increase the other pedestrians' trust in theintersection infrastructure warning system 1.

Similarly, the intersection infrastructure warning system 1 may use oneor more focused light displays 64 to output a focused light or focuseddisplay to warn the VRU 120. After the edge computing device 10 of theinfrastructure warning system 1 identifies the vehicle 130 posing a riskto the VRU 120, the edge computing device 10 outputs a visual warningformat and targeting instructions to the focused light display 64. ThePT device 55 (or the pan/tilt portion of an integrated focused lightdisplay design) then rotates and tilts the focused light display 64 toaim the output toward the VRU 120. The focused light display 64 thenoutputs a targeted (i.e., focused) visual warning 65 to the targeted VRU120.

For example, the visual warning 65 of the focused light display 64 a maybe a focused beam of light 65 a directed toward the eyes or face of thetargeted VRU 120 to get the attention of the targeted VRU 120 and warnthe VRU 120 of the vehicle 130.

In another example, the visual warning 65 of the focused light display64 b may be a holographic image of a stop sign 65 b instructing thetargeted VRU 120 to stop to avoid walking into the path of the vehicle130.

Similar to the local personalization of the holosonic audio warning 63,the focused visual warnings 65 may be visible only at a specificlocation. That is, the focused visual warnings 65 may only be visible atthe location of the VRU 120 to only alert the targeted VRU 120. In thismanner, other pedestrians at the intersection 100 may not be able to seethe warnings 65. Consequently, the other pedestrians are not erroneouslywarned of danger, which can increase the other pedestrians' trust in theintersection infrastructure warning system 1.

While the example intersection infrastructure warning system 1 in FIG.10 shows one holosonic speaker 62 and two focused light displays 64 aand 64 b, any number and combination of speakers 62 and displays 64 maybe used by the intersection infrastructure warning system 1.

The intersection infrastructure warning system 1 may also employ all ofthe holosonic speaker 62, the focused light display 64, and the drone 70as warning devices 60 to warn a targeted VRU 120 of danger. Multiplewarning devices 60 and RSUs 20 may be deployed at complex intersectionswhere multiple high risk scenarios are regularly present in order towarn a greater number of VRUs simulataneously.

While the previous example embodiments in FIGS. 7-10 all describesituations where the VRUs 120 were warned at the intersection 100, forexample, a VRU 120 crossing at designated crosswalks 106, theintersection infrastructure warning device 1 may also be configured toprovide warnings further away from the intersection.

With reference to FIG. 11, the example intersection infrastructurewarning system 1 at intersection 100 includes warning a VRUs 120 at ahundred meter distance from the intersection 100 using a drone 70 withfurther risk remediation by the drone 70. For example, in the examplesituation shown in FIG. 11, the sensor array 50 may be configured totrack and acquire data for the VRU 120 up to distances of two hundredmeters from the intersection 100. In this example, the VRU 120 is ajaywalker crossing the road 101 one hundred meters away from theintersection 100. For example, a school located directly across the road101 from the VRU 120 may cause the VRU 120 to jaywalk in lieu walking afurther one hundred meters to the intersection 100 to cross the road 101at the crosswalk 106, and then walking another one hundred meters on theother side of the road 101 to arrive at the school. School children mayfind the extra walking to the intersection 100 to cross the road 101 atthe crosswalk 106 to be inconvenient and bothersome. Moreover, if manyschool children jaywalk on the road 101, it may make other childrenthink that jaywalking is acceptable and encourage a greater number ofchildren to jaywalk on the road 101. Young school children may notappreciate the short time it may take for a vehicle 130 to travel onehundred meters. For example, if the vehicle 130 is speeding and drivingat a speed of fifteen mph over a forty-five mph speed limit (i.e., sixtymph), it may take the vehicle 130 approximately four seconds to travelone hundred meters. A vehicle 130 traveling the speed limit offorty-five mph may travel one hundred meters in about five seconds.Young children may not appreciate or understand speed and distance andhave a false sense that a speeding vehicle 130 is at a safe distance forcrossing the road 101. As such, the intersection infrastructure warningsystem 1 may also be used to provide warning to jaywalking VRUs 120crossing roads away from the intersection 100.

In the example of FIG. 11, the edge computing device 10 may predict thatthe VRU 120 will jaywalk across the road 101, while also determiningthat the vehicle 130 will not stop or slow down for the jaywalking VRU120. In this case, the edge computing device 100 may predict that theVRU 120 may walk into the path of the vehicle 130 causing a collisionbetween the VRU 120 and the vehicle 130.

After the edge computing device 10 of the infrastructure warning system1 identifies the vehicle 130 posing a risk to the VRU 120, the edgecomputing device 10 outputs respective warning message formats, deliverymodes, and flight instructions (i.e., response and targetinginstructions) to the drone 70.

The drone 70 then deploys from the nest 90, flies along the road 101,and drops to block the path of the VRU 120 trying to cross the road 101.That is, the drone 70 may fly from the nest 90 toward the VRU 120, drop,and then hover in front of the VRU 120 to block the walking path of theVRU 120. As in the above-described example embodiments, the warningoutput of the drone may be based on the threat level and the threatranking. In this example, the drone 70 may hover, avoiding contact withthe VRU 120, to block the path of the VRU 120 to prevent the VRU 120from jaywalking across the road 101.

The drone 70 may then attempt to direct the VRU 120 to the intersection100 to cross the road 101 at the crosswalk 106.

The RSU 20 may periodically output stored spatial-temporal data from thestorage 26 to the cloud 40 for further processing. In the example shownin FIG. 11, trained models and simulations by the cloud 40 can be usedby the intersection infrastructure warning system 1 to better determinepopular jaywalking locations on roads near the intersection 100. Trainedmodels from the cloud 40 can be used to better warn jaywalking VRUs 120.

The example intersection infrastructure warning system 1 in FIG. 11 mayalso be configured to deploy drones 70 to block all jaywalkingpedestrians, regardless of threat level, or absent any threat fromvehicles, to discourage jaywalking.

Too many false positive warning will adversely affect road users'responsiveness to a warning system. That is, a warning system outputtingtoo many false positive warnings will lead road users to eventuallyignore the warnings, thereby creating a riskier safety environmentaround an intersection. As such, success factors for the intersectioninfrastructure warning system 1 are warnings that are consistent andreliable.

The intersection infrastructure warning system including sensors, sensorfusion, and detection algorithms maintains cognitive recognition of thestate and behavior of all road users in the intersection vicinity byincluding sensors configured to acquire data 360° around theintersection. Using data captured by the road users, the GPU of a roadside unit analyzes the spatial-temporal data from road users with AIalgorithms to predict the trajectories, paths, behaviors, and intents ofroad users around the intersection. These predictions are furtheranalyzed by the GPU using AI algorithms to predict convergences betweenroad users that may cause imminent threat interactions between the roadusers. These threats can then be ranked so that warnings can beinitiated accordingly. One or more warnings can be deployed depending onthe threat level. That is, multiple warnings may be employedsimultaneously for different road users in different areas of anintersection to deter or prevent an accident. Alternatively, the warningdevices may be used to guide road users to positions of safety.

For each threat and threat level, the warning system 1 will output thesame warning response corresponding to a particular threat and threatlevel. In this way, consistent warning responses for each threat andthreat level conditions the road users to expect a response formitigating danger, thus increasing the safety effectiveness of theintersection infrastructure warning system 1. Reliability of theintersection infrastructure warning system 1 can be reinforced throughanomaly detection and mitigation. That is, data saved by the road sideunit can be output to the cloud for further modeling and simulation todetect and mitigate anomalies with the intersection infrastructurewarning system 1. Consequently, threat classification and anomalydetection and mitigation are considered in training the A algorithms.

While the preceding paragraphs describe the intersection infrastructurewarning system 1 as configured to warn VRUs and vehicles around anintersection of collisions, the intersection infrastructure warningsystem 1 is not limited to these functions. For example, theintersection infrastructure warning system 1 described herein could alsobe adapted for traffic management. That is, the intersectioninfrastructure warning system 1 could be at intersections in lieu ofpolice and traffic personnel to direct the flow of vehicle traffic,while also controlling the directing and controlling the flow ofpedestrian traffic.

In this manner, the drones 70 a and 70 b can respectively warn thetargeted VRUs 120 a and 120 b and also remedy dangerous situations bydirecting the targeted VRUs 120 a and 120 b away from the danger.

Example embodiments are provided so that this disclosure will bethorough, and will fully convey the scope to those who are skilled inthe art. Numerous specific details are set forth such as examples ofspecific components, devices, and methods, to provide a thoroughunderstanding of embodiments of the present disclosure. It will beapparent to those skilled in the art that specific details need not beemployed, that example embodiments may be embodied in many differentforms and that neither should be construed to limit the scope of thedisclosure. In some example embodiments, well-known processes,well-known device structures, and well-known technologies are notdescribed in detail.

The terminology used herein is for the purpose of describing particularexample embodiments only and is not intended to be limiting. As usedherein, the singular forms “a,” “an,” and “the” may be intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. The terms “comprises,” “comprising,” “including,” and“having,” are inclusive and therefore specify the presence of statedfeatures, integers, steps, operations, elements, and/or components, butdo not preclude the presence or addition of one or more other features,integers, steps, operations, elements, components, and/or groupsthereof. The method steps, processes, and operations described hereinare not to be construed as necessarily requiring their performance inthe particular order discussed or illustrated, unless specificallyidentified as an order of performance. It is also to be understood thatadditional or alternative steps may be employed.

Spatial and functional relationships between elements (for example,between modules, circuit elements, semiconductor layers, etc.) aredescribed using various terms, including “connected,” “engaged,”“coupled,” “adjacent,” “next to,” “on top of,” “above,” “below,” and“disposed.” Unless explicitly described as being “direct,” when arelationship between first and second elements is described in the abovedisclosure, that relationship can be a direct relationship where noother intervening elements are present between the first and secondelements, but can also be an indirect relationship where one or moreintervening elements are present (either spatially or functionally)between the first and second elements.

As used herein, the phrase at least one of A and B should be construedto mean a logical (A OR B), using a non-exclusive logical OR. Forexample, the phrase at least one of A and B should be construed toinclude any one of: (i) A alone; (ii) B alone; (iii) both A and Btogether. The phrase at least one of A and B should not be construed tomean “at least one of A and at least one of B.” The phrase at least oneof A and B should also not be construed to mean “A alone, B alone, butnot both A and B together.” The term “subset” does not necessarilyrequire a proper subset. In other words, a first subset of a first setmay be coextensive with, and equal to, the first set. As used herein,the term “and/or” includes any and all combinations of one or more ofthe associated listed items.

In the figures, the direction of an arrow, as indicated by thearrowhead, generally demonstrates the flow of information (such as dataor instructions) that is of interest to the illustration. For example,when element A and element B exchange a variety of information butinformation transmitted from element A to element B is relevant to theillustration, the arrow may point from element A to element B. Thisunidirectional arrow does not imply that no other information istransmitted from element B to element A. Further, for information sentfrom element A to element B, element B may send requests for, or receiptacknowledgements of, the information to element A.

In this application, including the definitions below, the term “module”or the term “controller” may be replaced with the term “circuit.” Theterm “module” may refer to, be part of, or include: an ApplicationSpecific Integrated Circuit (ASIC); a digital, analog, or mixedanalog/digital discrete circuit; a digital, analog, or mixedanalog/digital integrated circuit; a combinational logic circuit; afield programmable gate array (FPGA); a processor circuit (shared,dedicated, or group) that executes code; a memory circuit (shared,dedicated, or group) that stores code executed by the processor circuit;other suitable hardware components that provide the describedfunctionality; or a combination of some or all of the above, such as ina system-on-chip.

The module may include one or more interface circuits. In some examples,the interface circuit(s) may implement wired or wireless interfaces thatconnect to a local area network (LAN) or a wireless personal areanetwork (WPAN). Examples of a LAN are Institute of Electrical andElectronics Engineers (IEEE) Standard 802.11-2016 (also known as theWIFI wireless networking standard) and IEEE Standard 802.3-2015 (alsoknown as the ETHERNET wired networking standard). Examples of a WPAN arethe BLUETOOTH wireless networking standard from the Bluetooth SpecialInterest Group and IEEE Standard 802.15.4.

The module may communicate with other modules using the interfacecircuit(s). Although the module may be depicted in the presentdisclosure as logically communicating directly with other modules, invarious implementations the module may actually communicate via acommunications system. The communications system includes physicaland/or virtual networking equipment such as hubs, switches, routers, andgateways. In some implementations, the communications system connects toor traverses a wide area network (WAN) such as the Internet. Forexample, the communications system may include multiple LANs connectedto each other over the Internet or point-to-point leased lines usingtechnologies including Multiprotocol Label Switching (MPLS) and virtualprivate networks (VPNs).

In various implementations, the functionality of the module may bedistributed among multiple modules that are connected via thecommunications system. For example, multiple modules may implement thesame functionality distributed by a load balancing system. In a furtherexample, the functionality of the module may be split between a server(also known as remote, or cloud) module and a client (or, user) module.

Some or all hardware features of a module may be defined using alanguage for hardware description, such as IEEE Standard 1364-2005(commonly called “Verilog”) and IEEE Standard 1076-2008 (commonly called“VHDL”). The hardware description language may be used to manufactureand/or program a hardware circuit. In some implementations, some or allfeatures of a module may be defined by a language, such as IEEE1666-2005 (commonly called “SystemC”), that encompasses both code, asdescribed below, and hardware description.

The term code, as used above, may include software, firmware, and/ormicrocode, and may refer to programs, routines, functions, classes, datastructures, and/or objects. The term shared processor circuitencompasses a single processor circuit that executes some or all codefrom multiple modules. The term group processor circuit encompasses aprocessor circuit that, in combination with additional processorcircuits, executes some or all code from one or more modules. Referencesto multiple processor circuits encompass multiple processor circuits ondiscrete dies, multiple processor circuits on a single die, multiplecores of a single processor circuit, multiple threads of a singleprocessor circuit, or a combination of the above. The term shared memorycircuit encompasses a single memory circuit that stores some or all codefrom multiple modules. The term group memory circuit encompasses amemory circuit that, in combination with additional memories, storessome or all code from one or more modules.

The term memory circuit is a subset of the term computer-readablemedium. The term computer-readable medium, as used herein, does notencompass transitory electrical or electromagnetic signals propagatingthrough a medium (such as on a carrier wave); the term computer-readablemedium may therefore be considered tangible and non-transitory.Non-limiting examples of a non-transitory computer-readable medium arenonvolatile memory circuits (such as a flash memory circuit, an erasableprogrammable read-only memory circuit, or a mask read-only memorycircuit), volatile memory circuits (such as a static random accessmemory circuit or a dynamic random access memory circuit), magneticstorage media (such as an analog or digital magnetic tape or a hard diskdrive), and optical storage media (such as a CD, a DVD, or a Blu-rayDisc).

The apparatuses and methods described in this application may bepartially or fully implemented by a special purpose computer created byconfiguring a general purpose computer to execute one or more particularfunctions embodied in computer programs. The functional blocks andflowchart elements described above serve as software specifications,which can be translated into the computer programs by the routine workof a skilled technician or programmer.

The computer programs include processor-executable instructions that arestored on at least one non-transitory computer-readable medium. Thecomputer programs may also include or rely on stored data. The computerprograms may encompass a basic input/output system (BIOS) that interactswith hardware of the special purpose computer, device drivers thatinteract with particular devices of the special purpose computer, one ormore operating systems, user applications, background services,background applications, etc.

The computer programs may include: (i) descriptive text to be parsed,such as HTML (hypertext markup language), XML (extensible markuplanguage), or JSON (JavaScript Object Notation), (ii) assembly code,(iii) object code generated from source code by a compiler, (iv) sourcecode for execution by an interpreter, (v) source code for compilationand execution by a just-in-time compiler, etc. As examples only, sourcecode may be written using syntax from languages including C, C++, C#,Objective C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl,Pascal, Curl, OCaml, JavaScript®, HTML5 (Hypertext Markup Language 5threvision), Ada, ASP (Active Server Pages), PUP (PUP: HypertextPreprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, VisualBasic®, Lua, MATLAB, SIMULINK, and Python®.

The foregoing description is merely illustrative in nature and is in noway intended to limit the disclosure, its application, or uses. Thebroad teachings of the disclosure can be implemented in a variety offorms. Therefore, while this disclosure includes particular examples,the true scope of the disclosure should not be so limited since othermodifications will become apparent upon a study of the drawings, thespecification, and the following claims. Further, although each of theembodiments is described above as having certain features, any one ormore of those features described with respect to any embodiment of thedisclosure can be implemented in and/or combined with features of any ofthe other embodiments, even if that combination is not explicitlydescribed. In other words, the described embodiments are not mutuallyexclusive, and permutations of one or more embodiments with one anotherremain within the scope of this disclosure.

What is claimed is:
 1. An intersection infrastructure warning system forwarning vulnerable road users (VRUs) of danger around an intersection,the warning system comprising: an edge computing device having agraphics processing unit (GPU), a central processing unit (CPU), astorage device, and a communications module configured for wired orwireless communication, the edge computing device configured tocommunicate with a distributed cloud networking system; one or moresensors configured to acquire spatial-temporal data of road users,including the VRUs, in a 360° area around the intersection; and one ormore warning devices configured to output a warning to warn targetedVRUs of the danger around the intersection, wherein the GPU isconfigured to use one or more artificial intelligence (AI) algorithms toprocess the spatial-temporal data of the road users to determinepredictions of the road users, the predictions including at least one ofa path, a trajectory, a behavior, and an intent of a road user, the GPUis further configured to analyze the predictions of the road users todetermine convergences between the predictions of the road users todetermine threat interactions and identify the targeted VRUs, inresponse to determining threat interactions and identifying the targetedVRUs, the edge computing device is further configured to outputtargeting instructions and warning response instructions to the one ormore warning devices to deploy the one or more warning devices to thetargeted VRUs, and in response to receiving the targeting instructionsand warning response instructions, the one or more warning devices arefurther configured to target the targeted VRUs based on the targetinginstructions and to output the warning to the targeted VRUs.
 2. Thewarning system of claim 1, wherein the AI algorithms include at leastone of: an Isolated Forest model and a Sequence-to-sequence model. 3.The warning system of claim 1, wherein the one or more sensors includeat least one of: a camera and a detection and ranging sensor.
 4. Thewarning system of claim 3, wherein the at least one of the camera andthe detection and ranging sensor are configured as a sensor array. 5.The warning system of claim 1, wherein the one or more warning devicesinclude at least one of: a holosonic device, a focused light display, adrone, and a jetted fluid output device.
 6. The warning system of claim5, wherein the at least one of: the holosonic device, the focused lightdisplay, and the jetted fluid output device include a pan tilt deviceconfigured to rotate and tilt the at least one of: the holosonic device,the focused light display, and the jetted fluid output device to aim anoutput of the at least one of: the holosonic device, the focused lightdisplay, and the jetted fluid output device at the VRU.
 7. The warningsystem of claim 5, wherein the at least one holosonic device is aholosonic speaker configured to output an audio warning that is onlyaudible and discernible by a targeted VRU.
 8. The warning system ofclaim 5, wherein the at least one focused light display is configured tooutput a visual warning that is only visible and discernible by atargeted VRU.
 9. The warning system of claim 5, wherein the at least onedrone is configured to fly to a targeted VRU to at least one of: block apath of the targeted VRU, output an audio warning to the targeted VRU,output a visual warning to the targeted VRU, and guide the targeted VRUto a safe position.
 10. The warning system of claim 1, wherein the edgecomputing device uses the predictions of the road users to determinelocations of the targeted VRUs for deploying the one or more warningdevices.
 11. A method for warning vulnerable road users (VRUs) of dangeraround an intersection, the method comprising: identifying road usersaround the intersection; accumulating spatial-temporal data for each ofthe identified road users; processing the spatial-temporal data for eachof the identified road users to determine at least one of: trajectories,paths, intents, and spatial-temporal behaviors for each of theidentified road users; processing the at least one of: the trajectories,paths, intents, and spatial-temporal behaviors for each of theidentified road users using a first artificial intelligence (AI)algorithm to determine predictions for each of the identified roadusers, the predictions including at least one of: predictedtrajectories, predicted paths, predicted intents, and predictedspatial-temporal behaviors; and analyzing the predictions for each ofthe identified road users using a second AI algorithm to determine ifany conflict zones exist for any of the identified road users, theconflict zones indicating an interaction risk between any of theidentified road users.
 12. The method of claim 11 further comprising: inresponse to determining conflict zones exists, identifying (i) targetedVRUs in conflict zones, (ii) threats to the targeted VRUs in theconflict zones, and (iii) time horizons of the threats to the VRUs inthe conflict zones; prioritizing the threats to the targeted VRUs in theconflict zones; determining a warning response for the threats to thetargeted VRUs in the conflict zones; determining targeting instructionsto output a warning for each of the targeted VRUs; and outputting thetargeting instructions and the warning response to a warning device. 13.An intersection infrastructure warning system for warning vulnerableroad users (VRUs) of danger around an intersection, the warning systemcomprising: an edge computing device having a graphics processing unit(GPU), a central processing unit (CPU), a storage device, acommunications module configured for wired or wireless communication,and a distributed cloud networking system; one or more sensorsconfigured to acquire data for determining spatial-temporal data of roadusers, including the VRUs, in a 360° area around the intersection; andone or more drones configured to output a warning to warn targeted VRUsof the danger around the intersection, wherein the GPU is configured touse one or more artificial intelligence (AI) algorithms to process thespatial-temporal data of the road users to determine predictions of theroad users, the predictions including at least one of a path, atrajectory, a behavior, and an intent of a road user, the GPU is furtherconfigured to analyze the predictions of the road users to determineconvergences between the predictions of the road users to determinethreat interactions and identify the targeted VRUs, in response todetermining threat interactions and identifying the targeted VRUs, theedge computing device is further configured to output targetinginstructions and warning response instructions to the one or more dronesto deploy the one or more drones to the targeted VRUs, and in responseto receiving the targeting instructions and the warning responseinstructions from the edge computing device, the one or more drones arefurther configured to fly to the targeted VRUs based on the targetinginstructions and to output the warning to the targeted VRU based on thewarning response instructions.
 14. The warning system of claim 13,further comprising one or more nests for the one or more drones, the oneor more nests configured to dock the one or more drones when not inflight and to charge a battery of the one or more drones.
 15. Thewarning system of claim 13, wherein the warning output by the one ormore drones is a visual warning.
 16. The warning system of claim 13,wherein the warning output by the one or more drones is an audiowarning.
 17. The warning system of claim 13, wherein the warning outputby the one or more drones includes blocking a path of the targeted VRUs.18. The warning system of claim 13, wherein the one or more drones arefurther configured to guide the targeted VRUs to safe location foravoiding the danger around the intersection.
 19. The warning system ofclaim 13, wherein the at least one A algorithm includes at least one of:an Isolated Forest model and a Sequence-to-sequence model.
 20. Thewarning system of claim 13, wherein the one or more sensors include atleast one of: a camera and a detection and ranging sensor.